K-branching UIO sequences for partially specified observable non-deterministic FSMs

In black-box testing, test sequences may be constructed from systems modelled as deterministic finite-state machines (DFSMs) or, more generally, observable non-deterministic finite state machines (ONFSMs). Test sequences usually contain state identification sequences, with unique input output sequences (UIOs) often being used with DFSMs. This paper extends the notion of UIOs to ONFSMs. One challenge is that, as a result of non-determinism, the application of an input sequence can lead to exponentially many expected output sequences. To address this scalability problem, we introduce K-UIOs: K-UIOs that lead to at most K output sequences from states of M. We show that checking K-UIO existence is PSPACE-Complete if the problem is suitably bounded; otherwise it is in EXPSPACE and PSPACE-Hard. We provide a massively parallel algorithm for constructing K-UIOs and the results of experiments on randomly generated and real FSM specifications. The proposed algorithm was able to construct UIOs in cases where the existing UIO generation algorithm could not and was able to construct UIOs from FSMs with 38K states and 400K transitions

Message from the A-MOST 2021 Workshop Chairs

yesWe are pleased to welcome you to the 17th edition of the Advances in Model-Based Testing Workshop (A-MOST 2021), collocated with the IEEE International Conference on Software Testing, Verification and Validation (ICST 2021)

Accelerating finite state machine-based testing using reinforcement learning

Testing is a crucial phase in the development of complex systems, and this has led to interest in automated test generation techniques based on state-based models. Many approaches use models that are types of finite state machine (FSM). Corresponding test generation algorithms typically require that certain test components, such as reset sequences (RSs) and preset distinguishing sequences (PDSs), have been produced for the FSM specification. Unfortunately, the generation of RSs and PDSs is computationally expensive, and this affects the scalability of such FSM-based test generation algorithms. This paper addresses this scalability problem by introducing a reinforcement learning framework: the Q -Graph framework for MBT. We show how this framework can be used in the generation of RSs and PDSs and consider both (potentially partial) timed and untimed models. The proposed approach was evaluated using three types of FSMs: randomly generated FSMs, FSMs from a benchmark, and an FSM of an Engine Status Manager for a printer. In experiments, the proposed approach was much faster and used much less memory than the state-of-the-art methods in computing PDSs and RSs

Minimizing characterizing sets

A characterizing set (CS) for a deterministic finite state machine (FSM) M is a set of input sequences that, between them, separate (distinguish) all of the states of M. CSs are used within several test generation techniques that return test suites with guaranteed fault detection power. The number of input sequences in a CS directly affects the cost of applying the resultant test suite. In this paper, we study the complexity of decision problems associated with deriving a smallest CS from an FSM, showing that checking the existence of a CS with K sequences is PSPACE-complete. We also consider the length of a CS, which is the sum of the lengths of the input sequences in the CS. It transpires that the problem of deciding whether there is a CS with length at most K is NP-complete. Motivated by these results, we introduce a heuristic to construct a CS, from a deterministic FSM, with the aim of minimizing the number of input sequences. We evaluated the proposed algorithm by assessing its effect when used within a classical test generation algorithm (the W-method). In the evaluation, we used both randomly generated FSMs and benchmark FSMs. The results are promising, with the proposed algorithm reducing the number of test sequences by 37.3% and decreasing the total length of the test suites by 34.6% on average

Incomplete adaptive distinguishing sequences for non-deterministic FSMs

The increasing complexity and criticality of software systems has led to growing interest in automated test generation. One of the most promising approaches is to use model based testing (MBT), in which test automation is based on a model of the implementation under test (IUT), with much of the work concerning finite state machine (FSM) models. Many FSM-based test generation techniques use, possibly adaptive, sequences to check the state of the IUT. Of particular interest are adaptive distinguishing sequences (ADSs) because their use can lead to relatively small tests. However, not all systems possess an ADS. In this work, we generalise the notion of incomplete ADSs to non-deterministic partial and observable FSMs. We show that the problem of checking the existence of a set of k incomplete ADSs that separates every pair of states is PSPACE-hard. Further, we generalise the notion of invertible sequences to non-deterministic partial and observable FSMs and show how invertible sequences can be used to derive additional incomplete ADSs. We propose a novel algorithm to generate incomplete ADSs and describe the results of experiments that evaluated its performance. The results indicate that the proposed method can generate sequences to identify states of the IUT and is faster and can process larger FSMs than other existing methods